Multivariate emulation of computer simulators: model selection and diagnostics with application to a humanitarian relief model

We present a common framework for Bayesian emulation methodologies for multivariate-output simulators, or computer models, that employ either parametric linear models or nonparametric Gaussian processes. Novel diagnostics suitable for multivariate covariance-separable emulators are developed and techniques to improve the adequacy of an emulator are discussed and implemented. A variety of emulators are compared for a humanitarian relief simulator, modelling aid missions to Sicily after a volcanic eruption and earthquake, and a sensitivity analysis is conducted to determine the sensitivity of the simulator output to changes in the input variables. The results from parametric and nonparametric emulators are compared in terms of prediction accuracy, uncertainty quantification and scientific interpretability

Bayesian Optimal Design for Ordinary Differential Equation Models

Bayesian optimal design is considered for experiments where it is hypothesised that the responses are described by the intractable solution to a system of non-linear ordinary differential equations (ODEs). Bayesian optimal design is based on the minimisation of an expected loss function where the expectation is with respect to all unknown quantities (responses and parameters). This expectation is typically intractable even for simple models before even considering the intractability of the ODE solution. New methodology is developed for this problem that involves minimising a smoothed stochastic approximation to the expected loss and using a state-of-the-art stochastic solution to the ODEs, by treating the ODE solution as an unknown quantity. The methodology is demonstrated on three illustrative examples and a real application involving estimating the properties of human placentas

acebayes: An R Package for Bayesian Optimal Design of Experiments via Approximate Coordinate Exchange

We describe the R package acebayes and demonstrate its use to find Bayesian optimal experimental designs. A decision-theoretic approach is adopted, with the optimal design maximizing an expected utility. Finding Bayesian optimal designs for realistic problems is challenging, as the expected utility is typically intractable and the design space may be high-dimensional. The package implements the approximate coordinate exchange algorithm to optimize (an approximation to) the expected utility via a sequence of conditional one-dimensional optimization steps. At each step, a Gaussian process regression model is used to approximate, and subsequently optimize, the expected utility as the function of a single design coordinate (the value taken by one controllable variable for one run of the experiment). In addition to functions for bespoke design problems with user-defined utility functions, acebayes provides functions tailored to finding designs for common generalized linear and nonlinear models. The package provides a step-change in the complexity of problems that can be addressed, enabling designs to be found for much larger numbers of variables and runs than previously possible. We provide tutorials on the application of the methodology for four illustrative examples of varying complexity where designs are found for the goals of parameter estimation, model selection and prediction. These examples demonstrate previously unseen functionality of acebayes

Bayesian Optimal Design for Ordinary Differential Equation Models

Bayesian optimal design is considered for experiments where it is hypothesised that the responses are described by the intractable solution to a system of non-linear ordinary differential equations (ODEs). Bayesian optimal design is based on the minimisation of an expected loss function where the expectation is with respect to all unknown quantities (responses and parameters). This expectation is typically intractable even for simple models before even considering the intractability of the ODE solution. New methodology is developed for this problem that involves minimising a smoothed stochastic approximation to the expected loss and using a state-of-the-art stochastic solution to the ODEs, by treating the ODE solution as an unknown quantity. The methodology is demonstrated on three illustrative examples and a real application involving estimating the properties of human placentas

Gibbs optimal design of experiments

Bayesian optimal design of experiments is a well-established approach to planning experiments. Briefly, a probability distribution, known as a statistical model, for the responses is assumed which is dependent on a vector of unknown parameters. A utility function is then specified which gives the gain in information for estimating the true value of the parameters using the Bayesian posterior distribution. A Bayesian optimal design is given by maximising the expectation of the utility with respect to the joint distribution given by the statistical model and prior distribution for the true parameter values. The approach takes account of the experimental aim via specification of the utility and of all assumed sources of uncertainty via the expected utility. However, it is predicated on the specification of the statistical model. Recently, a new type of statistical inference, known as Gibbs (or General Bayesian) inference, has been advanced. This is Bayesian-like, in that uncertainty on unknown quantities is represented by a posterior distribution, but does not necessarily rely on specification of a statistical model. Thus the resulting inference should be less sensitive to misspecification of the statistical model. The purpose of this paper is to propose Gibbs optimal design: a framework for optimal design of experiments for Gibbs inference. The concept behind the framework is introduced along with a computational approach to find Gibbs optimal designs in practice. The framework is demonstrated on exemplars including linear models, and experiments with count and time-to-event responses

An approach for finding fully Bayesian optimal designs using normal-based approximations to loss functions

The generation of decision-theoretic Bayesian optimal designs is complicated by the significant computational challenge of minimising an analytically intractable expected loss function over a, potentially, high-dimensional design space. A new general approach for approximately finding Bayesian optimal designs is proposed which uses computationally efficient normal-based approximations to posterior summaries to aid in approximating the expected loss. This new approach is demonstrated on illustrative, yet challenging, examples including hierarchical models for blocked experiments, and experimental aims of parameter estimation and model discrimination. Where possible, the results of the proposed methodology are compared, both in terms of performance and computing time, to results from using computationally more expensive, but potentially more accurate, Monte Carlo approximations. Moreover, the methodology is also applied to problems where the use of Monte Carlo approximations is computationally infeasible